Managing  Risk  with  TSP 

by  David  R.  Webb 

Hill  Air  Force  Base 

One  of  the  most  important  aspects  of  applying  the  Team  Software  Process  (TSPSM)  to  software  projects  oft  any  size  is  the 
increased  success  of  identifying,  tracking,  and  mitigating  risk.  The  Mission  Planning  Software  Section  of  the  Software 
Tngineering  Division  of  Hill  Air  Force  Base  (TISHD),  has  found  the  TSP’s  simple  strategy  for  identifying,  tracking,  and 
handling  risks  to  be  extremely  effective.  In  fact,  many  common  software  project  risks  are  managed  purely  by  adopting  the  TSP. 


Dozens  of  books  address  the  concept 
of  software  risk  management  and,  it 
seems,  there  are  even  more  software  tools 
than  books  on  this  topic.  Risk  manage¬ 
ment  tools  can  be  as  simple  as  a  list  of 
risks  brainstormed  during  the  start  of  a 
project  and  reviewed  occasionally.  They 
can  also  be  as  complex  as  a  100-page  Risk 
Management  Plan  with  risks  and  their 
associated  prioritization,  likelihood, 
impacts  and  mitigation  strategies,  along 
with  a  Web-based  risk  browser  to  track 
the  plan.  Still,  the  basic  approach  for  all 
of  these  methods  is  the  same:  risks  are 
identified  early  in  the  project,  planned 
for,  monitored,  and  handled.  TSP  takes  a 
middle-of-the-road  approach  to  risk  man¬ 
agement,  doing  what  makes  sense  for  the 
project  with  as  little  paperwork  and  tool 
upkeep  as  possible.  Although  the 
approach  was  originally  designed  for 
teams  of  fewer  than  20  people,  the  prin¬ 
ciples  can  be  applied  to  much  larger 
groups,  with  equally  effective  results. 


Figure  1.  Risks  are  identified  at  each  TSP  launch. 
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Identification 

The  TSP  handles  a  project  the  way 
you  eat  an  elephant — one  bite  at  a  time. 
The  TSP  team  estimates  projects  in  a  top- 
down  approach,  using  overall  size  and 
average  team  productivity  to  determine 
overall  schedule.  This  schedule  is  broken 
into  manageable  phases  and  the  phase  cur¬ 
rently  being  worked  is  thoroughly  estimat¬ 
ed  and  tracked  using  a  bottom-up 
approach  wherein  each  engineer  estimates 
his  or  her  own  schedule  using  individual 
data.  Each  time  a  phase  begins,  whether  at 
the  start  of  the  project  or  at  the  transition 
from  one  phase  to  the  next,  there  is  a 
project  launch  (Figure  1).  At  these  launch¬ 
es,  the  tasks  for  the  current  phase  are  thor¬ 
oughly  defined  and  each  task  is  estimated 
using  the  rigorous  methods  of  the  Personal 
Software  Process  (PSPSM).  These  estimates 
are  used  to  produce  a  detailed  next  phase 
earned  value  plan,  against  which  the  proj¬ 
ect  will  be  tracked  and  managed.  Project 
goals,  quality  criteria  and  risks  are  also 
identified  during  the  launches. 

A  portion  of  each  launch  is  dedicated 
to  brainstorming  risks  the  project  may 
face.  These  sessions  can  last  from  a  dozen 
minutes  to  a  few  hours,  depending  upon 
the  size  of  the  project  and  the  team’s 
knowledge  and  maturity.  The  risks  that 
are  identified  are  serious  problems  that 
may  occur  during  the  life  cycle  of  the 
project,  not  just  a  list  of  all  maladies  that 
are  possible.  For  example,  it  makes  little 
sense  to  manage  the  risk  of  your  software 
being  destroyed  by  a  bomb  or  abducted 
by  aliens  unless,  of  course,  you  work  for 
Special  Agents  Fox  Mulder  and  Dana 
Scully.  Barring  that  circumstance,  most 
projects  make  a  list  of  all  the  real-life 
problems  that  can  be  foreseen.  Some  com¬ 
mon  risks  identified  during  these  meetings 
include  a  lack  of  proper  documentation,  a 
development  environment  that  may  not 
support  the  size  or  type  of  program  being 
developed,  an  impossible  schedule  or 


inadequate  computer,  office,  or  personnel 
resources.  Each  risk  is  assigned  a  likeli¬ 
hood  of  occurrence,  a  severity  if  it  does 
occur,  and  a  person  responsible  to  moni¬ 
tor  the  risk.  The  TSP  team  assigns  each 
member  a  role,  such  as  Design  Manager, 
Planning  Manager,  Implementation 
Manager,  Customer  Interface  Manager, 
Quality  Manager,  Process  Manager, 
Support  Manager,  Test  Manager,  or  Team 
Leader.  Typically,  the  team  member  with 
the  appropriate  role  is  assigned  to  monitor 
a  risk.  For  example,  a  risk  involving  nego¬ 
tiations  with  the  customer  would  be 
assigned  to  the  Customer  Interface 
Manager.  This  information  is  documented 
so  that  it  can  be  regularly  referenced. 

Review  and  Mitigation 

The  TSP  requires  a  weekly  status 
meeting  where  team  progress  is  compared 
to  the  team  plan  in  terms  of  earned  value 
and  quality.  If  there  are  deviations  from 
the  plan,  the  reasons  for  these  deviations 
can  be  determined  and  actions  taken  to 
bring  the  team’s  performance  in  line  with 
the  plan.  It  is  also  during  these  weekly 
meetings  that  the  team  reviews  the  risks 
brainstormed  during  the  launch.  The  team 
removes  risks  from  the  list  that  no  longer 
pose  a  threat,  while  the  assigned  engineers 
report  on  those  that  are  still  potential 
problems.  If  the  mitigation  strategy  for  a 
risk  has  failed  and  the  risk  has  occurred,  or 
is  likely  to  occur  soon,  the  risk  is  renamed 
an  “issue”  and  immediate  action  is  taken 
to  address  it.  The  risk  list  subsequently 
becomes  a  living,  breathing  document  that 
changes  size  and  shape  each  week.  It  also 
becomes  a  used  document  that  helps  the 
team  focus  on  risks  that  need  to  be 
addressed  when  they  need  to  be  addressed. 

Risks  That  Are  Not 

Three  of  the  most  common  risks  to 
any  project  are  schedule  overruns, 
requirements  creep,  and  quality  problems. 
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Figure  2.  TSP  Team  Earned  Value  for  TISHD  A- 10  AWE 


A  project  properly  using  the  TSP  already 
has  the  tools  to  handle  these  risks. 

A  TSP  team  determines  its  own 
schedule  and  coordinates  it  with  manage¬ 
ment,  marketing,  and  the  customer,  as 
appropriate.  While  outside  influences 
may  have  strong  impacts  on  the  delivery 
date  of  any  piece  of  software,  the  TSP 
team  knows  its  productivity  rates,  has  a 
rigorous  estimating  process,  and  can  con¬ 
fidently  tell  management  how  much  can 
be  accomplished  within  a  given  time 
frame.  The  TSP  launch  is  not  successfully 
concluded  until  the  team  and  manage¬ 
ment  agree  upon  a  list  of  requirements 
and  a  schedule  that  is  satisfactory  to  both 
parties.  Once  this  realistic  schedule  has 
been  determined,  it  is  used  as  the  basis 
for  measuring  personal  and  team-earned 
value  and  is  tracked  daily  at  the  personal 
level  and  weekly  at  the  team  level  (Figure 
2).  Any  deviations  from  the  plan  are 
identified  early  in  the  project  and  are 
dealt  with  by  negotiating  with  manage¬ 
ment  and  the  customer.  The  TSP  virtual¬ 
ly  eliminates  schedule  risks. 

The  TSP  also  requires  replanning,  or 
at  least  updating,  a  project  when  the  basic 
assumptions  of  the  plan  change.  This 
means  that  when  (not  if)  the  requirements 
change  during  the  course  of  the  project, 
the  team  renegotiates  schedule,  delivered 
functionality  and,  if  appropriate,  cost.  This 
becomes  the  new  plan  that  the  team  tracks 
and  the  requirements  creep  risk  is  effec¬ 
tively  dealt  with,  if  not  completely  elimi¬ 
nated.  Another  great  thing  about  this  tech¬ 
nique  is  that  it  ensures  management  and 
the  customer  are  involved  every  step  of  the 
way  so  that  no  one  is  surprised  by  the  pro¬ 
ject’s  performance,  least  of  all  those  who 
are  anticipating  the  product. 

Finally,  problems  with  quality  can, 
over  time,  be  virtually  erased  using  the 
TSP.  Since  the  quality  methods  used  by 
the  TSP  are  based  upon  the  strict  quality 
processes  of  the  Personal  Software  Process, 
individual  engineers  perform  their  own 
extensive  reviews  of  both  detailed  design 
and  code  prior  to  exhaustive  team  inspec¬ 
tions.  Defect  densities  at  personal  reviews, 
team  inspections,  compile  and  unit  test, 
are  used  as  yardsticks  to  determine  if  the 
finished  code  is  of  high  enough  quality  to 
be  passed  on  to  integration  and  system 
test,  or  if  the  code  should  be  pulled  back 
and  reinspected  or  rewritten.  This  ensures 


the  quality  of  the  code,  but  not  always  the 
quality  of  the  requirements  upon  which 
the  code  is  based.  Often,  requirement 
problems  are  uncovered  during  acceptance 
testing.  When  such  defects  are  discovered, 
the  TSP  team  adds  them  to  team  and  per¬ 
sonal  review  checklists  to  ensure  such 
problems  are  never  allowed  to  pass 
through  the  process  again.  An  experienced 
TSP  team  can,  therefore,  eliminate  virtu¬ 
ally  all  quality  risks,  particularly  expensive 
defects  found  during  qualification  and 
acceptance  testing. 

Some  Examples  in  TISHD 

TISHD  has  trained  nearly  20  engi¬ 
neers  in  the  PSP  and  has  launched  three 
separate  mission  planning  projects  using 
TSP  version  0.3.  These  projects  are  an  Air 
Tasking  Order  parser  named  TaskView  [1], 
an  A- 10  Aircraft/Weapons/Electronics 
(AWE)  software  program,  and  an  F-16 
Block  30  AWE  program.  Of  these  three 
projects,  TaskView  and  A- 10  AWE  have 
been  using  the  TSP  long  enough  for  us  to 
draw  some  conclusions  about  the  useful¬ 
ness  of  the  TSP  and  the  success  rate  of 
using  the  TSP  risk  management  strategy. 

TaskView 

The  TaskView  project  was  the  first 
TISHD  group  to  pilot  test  the  TSP.  Just 
prior  to  the  initial  launch,  the  TaskView 
customer  decided  to  participate  in  an  Air 


Force  Expeditionary  Force  Experiment 
(EFX).  This  new  goal  required  the 
TaskView  3.0  product  to  be  delivered  one 
month  earlier  than  originally  planned. 
The  team  added  this  risk  to  its  risk  list 
and  assigned  it  to  the  Planning  Manager. 
With  this  in  mind,  the  team  adjusted  the 
plan  to  meet  the  new  schedule. 

As  work  progressed,  the  team-earned 
value  projected  that  TaskView  3.0  would 
be  delivered  more  than  a  month  earlier 
than  anticipated,  even  with  the  new 
schedule.  At  this  point,  the  first  risk  was 
closed  out  and  another  risk — that  of 
being  too  early  and  losing  revenue — was 
added  to  the  list  and  assigned  to  the 
Customer  Interface  Manger.  The  cus¬ 
tomer  was  approached  with  the  option  of 
receiving  the  product  early  and  getting  a 
refund,  or  adding  new  capability  to 
TaskView  3.0.  The  customer  was  delight¬ 
ed  with  this  information  and  chose  to 
keep  the  current  level  of  funding  and  add 
in  new  capabilities  to  the  software.  Even 
with  the  new  functionality,  TaskView  3.0 
was  delivered  well  within  time  to  partici¬ 
pate  in  the  EFX  experiment. 

TaskView  has  also  experienced  a  sig¬ 
nificant  reduction  in  the  risks  associated 
with  defects,  as  a  result  of  adopting  the 
TSP.  TaskView  has  had  three  major  releas¬ 
es  since  TISHD  started  working  on  the 
project  in  1997.  TISHD  has  added  new 
capability  and  robustness  to  each  release, 
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at  times  rewriting  major  portions  of  the 
code  to  do  so.  Compared  to  data  from 
similar  projects  completed  in  the  past 
(using  the  TISHD  CMM  Level  5  organi¬ 
zational  process),  the  TaskView  projects 
have  seen  a  substantial  decrease  in  defects 
and  test  time  (see  Figure  3  and  Figure  4). 

One  interesting  outcome  of  the 
defect  data  analysis  was  the  increase  in 
defect  density  experienced  by  the 
TaskView  3.1  project.  Although  the  defect 
density  found  during  Customer 
Acceptance  Testing  was  steadily  decreas¬ 
ing,  defects  found  in  earlier  test  phases 
increased.  This  was  of  some  concern  to 
the  team,  until  it  began  to  filter  the  list  by 
defect  priority  (Figure  4).  Once  that  was 
done,  it  was  obvious  that  the  TaskView 
team,  as  it  had  grown  more  confident  in 
the  use  of  the  TSP,  had  begun  to  record 
more  development  defects  than  ever 
before;  remember,  TSP  teams  count  every 
defect  found  in  every  development  and 
test  phase,  including  compile.  However, 
despite  this  increase  in  defect  recording, 
high-priority  defects  became  nonexistent 


using  the  TSP.  This  does  not  mean  that 
no  issues  were  discovered  during  customer 
acceptance  testing,  but  the  issues  dealt 
almost  exclusively  with  the  addition  of 
new  requirements  and  limitations  of  the 
operational  environment,  and  were  not 
defects  in  the  delivered  code.  Note  also 
that  TaskView  3. IS  (a  special  project 
developed  in  support  of  another  mission 
planning  tool)  had  zero  high-priority 
defects  at  every  test  phase. 

As  most  software  project  managers  are 
well  aware,  the  greatest  risk  to  any  pro¬ 
ject’s  schedule  is  the  risk  of  finding  defects 
during  test,  especially  final  or  customer 
acceptance  testing.  The  causes  of  these 
defects  are  often  difficult  to  trace  and  fix 
and  can  cause  significant  slips  in  schedule. 
In  order  to  eliminate  this  risk,  test  time 
needs  to  be  reduced  and  become  more 
consistent.  Although  TISHD  was  already 
seeing  very  low  test  days/ thousand  lines  of 
code  rates  using  its  Level  5  process,  the 
adaptation  of  the  TSP  reduced  the  test 
time  further  and  made  the  variation  much 
smaller  (see  Figure  5). 


A-10  AWE 

While  the  TaskView  project  was  still 
evaluating  the  effectiveness  of  the  TSP,  the 
A-10  AWE  team  decided  to  use  some  of 
the  concepts  (planning,  tracking,  weekly 
updates)  without  employing  the  rigorous 
techniques  of  the  Personal  Software 
Process.  Each  A-10  AWE  engineer  was 
provided  a  spreadsheet  for  each  code 
change  he  or  she  was  working.  These 
spreadsheets  covered  the  estimate  of  size 
(lines  of  code  or  LOC)  and  time  (days)  as 
well  as  the  actuals  for  LOC  and  time. 

Time  was  measured  at  distinct  milestones, 
such  as  inspections,  unit  test,  and  code 
check-in.  An  earned  value  plan  was  created 
from  the  estimates  provided  by  the  engi¬ 
neers  and  used  to  refine  the  schedule. 
Although  the  engineers  were  not  required 
to  be  PSP  trained,  all  any  engineer  had  to 
do,  after  estimating,  was  to  check  a  box  on 
the  tracking  spreadsheet  once  a  milestone 
was  reached.  The  spreadsheet  would  calcu¬ 
late  how  long  the  tasks  took  and  export 
that  data  to  the  earned  value  tracking  tool. 

Sounds  like  a  good  plan,  right?  It  did 
not  work  very  well. 

Estimates  were  often  wildly  inaccu¬ 
rate.  Tracking  was  not  consistent.  Entire 
new  capabilities  would  move  from  0  per¬ 
cent  complete  to  100  percent  complete 
overnight.  All  of  these  problems  gave  the 
team  a  false  impression  of  the  team-earned 
value.  The  earned  value  was,  therefore,  not 
trusted  and  soon  ignored  by  most  of  the 
team  members.  The  team  reverted  to  the 
higher-level  tracking  process  used  by  non- 
TSP  projects  in  TISHD,  which  were  suffi¬ 
cient  to  prevent  the  team  from  missing 
schedule.  (Note  that  TISHD  is  part  of  a 
SW-CMM®  Level  5  organization,  and 
typically  meets  cost  and  schedule  estimates 
anyway.)  However,  any  advantage  of  using 
the  TSP-like  process  disappeared. 

The  one  thing  that  did  work,  was 
risk  identification  and  tracking,  the 
process  that  we  copied  directly  from  the 
Team  Software  Process. 

For  example,  the  A-10  AWE  team 
determined  that  a  required  piece  of  core 
software,  developed  by  a  third-party  ven¬ 
dor,  might  not  be  released  in  time  to 
meet  schedule.  This  risk  was  assigned  a 
high  likelihood  and  a  high  impact.  A  mit¬ 
igation  plan  of  reverting  to  an  earlier 
release  of  the  core  was  determined  and  an 
engineer  was  assigned  to  track  the  status 
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of  the  core  software.  As  it  turned  out,  the 
third-party  software  did  slip  its  schedule 
by  several  months,  which  would  have,  in 
turn,  caused  our  software  to  slip  its 
release  date  had  we  not  planned  for  this 
risk  early  in  the  program.  Due  to  early 
risk  identification,  planning  and  tracking, 
the  A- 1 0  AWE  was  able  to  mitigate  this 
risk  and  revert  to  the  earlier  version  of 
the  core  software. 

One  risk  that  was  not  identified  was 
the  hazard  of  using  the  TSP-like  process, 
instead  of  the  TSP.  During  project  post¬ 
mortem,  it  was  determined  that  the  reason 
the  modified  process  did  not  work  as  well 
as  a  traditional  TSP  team  was  that  the 
engineers  were  not  PSP  trained  and  did 
not  understand  how  the  data  they  were 
collecting  was  being  used.  At  that  point, 
we  determined  to  use  TSP  on  the  next  A- 
10  AWE  project  and  immediately  sched¬ 
uled  a  PSP  course  for  those  engineers. 

The  results  in  earned  value  tracking 
alone  were  astounding  (see  Figure  3). 

Code  was  accurately  estimated  and 
tracked;  it  was  very  easy  to  see  how  close  to 
our  schedule  we  were  running.  TISEID 
learned  an  important  lesson:  TSP  does  not 
work  well  without  the  proper  data,  and 
that  data  is  almost  impossible  to  gather 
without  the  rigors  of  the  PSP.  That  is  one 
risk  TISEID  has  completely  eliminated. 

Conclusion 

While  there  are  many  tools  for  soft¬ 
ware  risk  management,  TISHD  has 
found  that  utilizing  the  planning,  track¬ 
ing,  and  defect  prevention  techniques  of 
the  Team  Software  Process  is  a  simple  and 
effective  way  to  identify,  track,  and  miti¬ 
gate  most  software  project  risks.  In 


TISHD  we  have  learned  that,  over  time, 
TSP  teams  become  experts  at  risk  mitiga¬ 
tion  and  management;  they  also  become 
very  good  at  writing  code  that  is  nearly 
free  of  defects,  and  that  TSP  is  a  risk  mit¬ 
igation  strategy  any  software  project 
should  strive  to  adopt. 
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